Method for smoothing transitions between scenes of a stereo film and controlling or regulating a plurality of 3d cameras

ABSTRACT

In an exemplary embodiment, a method for producing a stereo film is provided, wherein a first image that is supplied ( 10 ) by a first camera rig having at least two cameras is followed ( 50 ) by a second image from a second camera rig, wherein furthermore a disparity table for definition of the displacement of a defined image point in a first sub-frame supplied by a first camera of the first camera rig relative to an image point similar thereto in a second sub-frame supplied by a second camera of the first camera rig is determined ( 20, 30 ) in order to obtain information about the depth of the first image composed of the first sub-frame and the second sub-frame, wherein the depth information of the disparity table of the first image of the first camera rig is used ( 60 ) for processing of the second image of the second camera rig. The invention also relates to controlling (means) or regulating means for a plurality of 3D cameras configured to carry out said method.

INCORPORATION BY REFERENCE TO ANY PRIORITY APPLICATIONS

Any and all applications for which a foreign or domestic priority claimis identified in the Application Data Sheet as filed with the presentapplication are hereby incorporated by reference under 37 CFR 1.57.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flowchart of a method for smoothing transitions betweenscenes of a stereo film and controlling or regulating a plurality of 3Dcameras.

FIG. 2 shows a schematic view of a system for smoothing transitionsbetween scenes of a stereo film and controlling or regulating aplurality of 3D cameras.

DETAILED DESCRIPTION

This application relates to a method for producing a stereo film,wherein a first image that is supplied by a first camera rig having atleast two cameras is followed by a second image from a second camerarig, wherein furthermore a disparity table for the definition of thedisplacement of a defined image point in a first sub-frame supplied by afirst camera of the first camera rig relative to an image point similarthereto in a second sub-frame supplied by a second camera of the firstcamera rig is determined in order to obtain information about the depthof the first image composed of the first sub-frame and the secondsub-frame.

Camera rigs and methods for producing 3D films, or so-called stereofilms, are known from prior art. Such films project a specific image foreach eye of the viewer so that a three dimensional image is composed forthe viewer.

Usually for the case of camera rigs used for the recording of scenes,two cameras are combined in each camera rig. While a first camera righaving two cameras is directed from a first viewing angle towards ascene to be filmed, a second camera having two further combined camerasis directed at a different viewing angle towards the scene. If now a“cut” is performed from the first camera rig to the second camera rig,i.e. if a sequence of images follows, which images are supplied by thefirst camera rig, said sequence being a sequence of images following thecut, which sequence is supplied by the other camera rig, then, in thecase of three dimensional films, undesired effects often occur for theviewer due to the cut.

Thus it can happen that, when the first camera rig is directed towards ascene, a much larger impression of depth is achieved such that forexample an object of the scene is perceived by the viewer to be far infront of a virtual plane of the screen, whereas upon cutting, the objectis perceived as being far behind the plane of the screen, or at leastnot at the position observed from the other perspective a short whilepreviously.

While no unpleasant effects occur in real life, if an object shouldsuddenly come towards the viewer, this fast change in impression ofdepth during the reproduction of related information leads to discomfortfor the viewer.

This is, inter alia, because in real life, for example in the case ofobserving a landscape, wherein an object, such as a ball spontaneouslyspeeds towards a viewer, this ball is not in focus at first and thus thenegative effects that occur when observing a film do not appear.

Therefore it is an object of some embodiments of the present inventionto allow a cut to be made in a stereo film such that a first image thatis supplied by a first camera rig having at least two cameras can befollowed by a second image from a second camera rig, and the impressionof depth created in both images does not cause unpleasant side effectsfor the viewer.

This object is solved according to some embodiments of the presentinvention in that the depth information of the disparity table of thefirst image of the first camera rig is used for the processing of thesecond image of the second camera rig.

A disparity table is understood to mean such a compilation ofinformation that makes possible the assessment of the impression ofdepth of the first image. The impression of depth is created by means ofan image depth analysis which can also be called lateral disparity ordisparity. Such a disparity is an offset in the position which the sameobject in the picture occupies on two different image planes. Theoptical centers for the image planes of the lenses are in this wayspatially separated from each other by the basis b. If both lenses havethe focal length f,

$r = \frac{b \cdot f}{d}$

applies for distance r, wherein d stands for the disparity. This formulaapplies only to the stereo normal case, i.e. when the two cameras arealigned in parallel. If both cameras are slightly pivoted towards oneanother, i.e., convergently aligned, a modified formula is applicable.

One can therefore determine the distance r to an object by a measuringof the disparities in the stereo image. A disparity map or a disparitytable of a stereo image is therefore synonymous with a depth map.

It should be noted here that an image is understood to mean thecompilation of two sub-frames, wherein each of the two sub-frames issupplied by one of the two cameras of a defined camera rig.

Some embodiments of the invention, which can be implemented in aStereoscopic Image Processor (SIP), can analyze a scene and providemetadata concerning the depth or depth information of a near and distantobject, and also supply information regarding the total space/totalvolume in real-time. In addition the SIP can also perform an imageprocessing and image manipulation. In some embodiments of the proposedinvention it is achieved that, based on the provided data, it is ensuredthat 3D-changes within a scene remain within the depth budget.

Advantageous embodiments are claimed in the dependent claims and areexplained in more detail below.

Thus it is advantageous when a second sub-frame is displaced relative toa first sub-frame supplied from a first camera of the two cameras,wherein said first sub-frame forms the second image together with thesecond sub-frame supplied from a second camera of the second camera rig.With a displacement of both sub-frames of the second camera rig relativeto each other, the impression of depth is changed. Therefore theimpression of depth can be adjusted to fit the impression of depth inthe previously present first image.

When the second sub-frame is displaced horizontally, one can resort to astandard depth effect generation. If the second sub-frame, i.e. if forexample a right sub-frame, is displaced from a left sub-frame towardsthe right, then the depth effect increases, whereas the depth effect isreduced for the reverse case. This is due to the lines of sight whichrun almost parallel to each other when observing a distant object,whereas in the case of a very near object, even an intersecting of thesightlines can occur. The disparities are arranged around a zero point,and can thus take negative as well as positive values.

It is further advantageous when the second sub-frame of the secondcamera rig is displaced in a displacement step by such a distance untilthe same disparity is present between the two sub-frames of the secondimage as between the two sub-frames of the first image. The viewer's eyein that case does not have to adjust and negative effects are almostcompletely removed.

The method can be further improved if the second image is magnified orreduced by means of a zoom setting with a correction step dependent onthe disparity table of the first image, until the depth distance betweena point in the foreground and a point in the background of the secondimage corresponds to the depth distance between these two points in thesecond image. By means of the change of the zoom setting, the perceiveddepth distance from a first object in the scene to a second object inthe scene changes.

The disparities also diminish when the zoom setting operates only as adigital zoom and does not operate mechanically on the physical lenses ofthe second camera rig.

It is a further advantage here when the first and second sub-frame ofthe second image is magnified or reduced. With a reduction of the secondimage, the disparities also reduce linearly with the reduction of theimage so that the negative side effects caused by disparities being toohigh when cutting do not appear to the user.

When the correction step is performed simultaneously with or followingthe displacing step, a positive result can be achieved particularlyquickly in the first of the two cases, whereas a particularly exactresult can be achieved in the second case.

It is further advantageous when the depth budget which is placed in thedisparity table is applied to the second image such that all areas ofthe second image which lie beyond the depth budget are shown blurred. Adepth budget is understood to mean the range caused by thedisparities/the region caused by the disparities. Therefore when thesmallest disparity is, for example, −50 and the largest disparity is 0,the image comprises a depth budget of −50 to 0.

In order to achieve a particularly efficient blurring, a Gaussianblurring algorithm is used to achieve the blurring in one or a pluralityof regions of the second image. Therefore the areas are identified inwhich the depth budget of the second image is too large in comparison tothe depth budget in the first image, and the identified areas are thendisplayed blurred. Under the Gaussian blurring algorithm, thesurrounding pixels are used and the pixels which are to be displayedexcessively blurred are recalculated according to a Gaussian normaldistribution.

Some embodiments of invention also relate to a controlling or regulatingof a plurality of 3D cameras, said 3D cameras being suited for therecording of a stereo film, wherein the controlling or regulating isconfigured such that it can perform the method according to theinvention.

Some embodiments of the invention are also subsequently explained inmore detail with the help of a picture. A first exemplary embodiment isvisualized in a schematically depicted flowchart of a first figure (FIG.1), wherein a second exemplary embodiment is visualized in a furtherfigure (FIG. 2).

FIG. 1 shows a flowchart of a first exemplary embodiment of a methodaccording to the invention:

In a first step 10, the recording of a first image takes place with afirst camera rig comprising two cameras. In a subsequent second step 20,the disparities in the first image are ascertained.

In a subsequent third step 30, the setting up of a disparity tableoccurs, which can also be described as a disparity map.

In step 40, a cut takes place from the first camera rig to the secondcamera rig during the production of the film sequence of the stereofilm. The second camera rig also contains two cameras.

In the subsequent step 50, the recording of a second image takes placewith the second camera and its two cameras.

In the subsequent step 60, the use of the disparity table for theprocessing of the second image takes place.

In a sub-step 61, which is followed by a subsequent step 62 in theexemplary embodiment depicted here, the displacing of a sub-frame of thesecond image to another sub-frame of the second image takes place,wherein both of these two sub-frames form in total the second image. Inthe sub-step 62, a correction step is performed, in other words the zoomsetting in the second image is changed. Thus in the displacing step 61,the disparity distribution in total is changed, whereas in thecorrection step 62 the present disparities per se are changed.

A blurring step 63 can also be performed in parallel to, subsequent to,or as an alternative to this. The blurring step comprises an identifyingof areas of the second image which have too large or too small adisparity compared to the disparities of the first image. A blurring ofthese areas is realized, for example, by means of a Gaussian blurringalgorithm.

A schematic construction of a second exemplary embodiment according tothe invention is depicted in the second figure (FIG. 2):

Two cameras 110 and 120 are contained in each camera rig 100, whichcameras send image data of a stereoscopic image pair to an imageanalysis unit 130. The image analysis unit 130 determines the scenedepth in terms of near, middle and of a more remote area in real time.

These obtained metadata are either embedded in the image data of one orthe other image of the stereoscopic image pair, or embedded in the imagedata of both images. These processed data are passed on to a switch 140,also identified as switcher.

The switch 140 allows a user to choose among the source data for anoutput interface 160, under the interposition of an image processor 150.The image processor 150 contains statistical depth budget parameters, inparticular background data relating to a maximum allowable change perunit of time.

The image processor 150 manages a dynamic statistic from the depthinformation obtained from the metadata, calculates rates of change andchange magnitudes and ensures that these values are within the depthbudget to be used.

If the resulting depth change lies within the predetermined envelope,then the image pair is passed on unchanged to the output interface 160.If this is done repeatedly one after the other, the result is a videosequence.

If the depth change is not contained within the predefined depth budget,an image is adjusted, for example blurred, smudged, obscured,desaturated and/or masked/marked. The bigger the deviation from thedepth budget, then the bigger the correcting operation of the blurring,for example. The operation can also include a blackflash or whiteflash,in other words a transition from black or white to some new imagecontent. Instead of processing only a defined area, the entire image ofthe image pair can be blurred. This is particularly advantageous when noexact disparity map is available.

The method is repeated for each stereoscopic image pair.

1. A method for producing a motion video recording, comprising:obtaining a first image captured with a first camera rig having at leasttwo cameras; obtaining a second image captured with a second camera rig;calculating, with an image processor, a first displacement of at leastone first image point contained in a first sub-frame obtained from afirst camera of the first camera rig relative to at least one secondimage point contained in a second sub-frame obtained from a secondcamera of the first camera rig and corresponding to the at least onefirst image point; storing the at least one displacement in a disparitytable; and processing the at least one second image using the disparitytable.
 2. The method of claim 1, further comprising displacing thesecond sub-frame relative to the first sub-frame, and wherein the firstsub-frame and a fourth sub-frame obtained from a second camera of thesecond camera rig form the second image.
 3. The method of claim 2,wherein the second sub-frame is horizontally displaced.
 4. The method ofclaim 1, further comprising displacing a fourth sub-frame obtained froma second camera of the second camera rig relative to a third sub-frameobtained from a first camera of the second camera rig to create a seconddisplacement.
 5. The method of claim 1, further comprising changing themagnification of the at least one second image in response to thedisparity table and until a second depth distance between a secondforeground point in a third sub-frame of the second image and a secondbackground point in a fourth sub-frame of the second image issubstantially the same as a first depth distance between a firstforeground point in the first sub-frame and a first background point inthe second sub-frame.
 6. The method of claim 5, wherein themagnification is changed digitally.
 7. The method of claim 5, whereinthe magnification of both the third sub-frame and the fourth sub-frameis changed.
 8. The method of claim 2, wherein the processing the atleast one second image is performed simultaneously with displacing thesecond sub-frame relative to the first sub-frame.
 9. The method of claim1 further comprising blurring all areas of the second image that lieoutside a depth budget of the disparity table.
 10. The method of claim9, wherein the blurring is accomplished using a Gaussian blurringalgorithm.
 11. (canceled)
 12. The method of claim 1, wherein thedisplacement corresponds to a depth of the first image.
 13. The methodof claim 4, wherein the second displacement is substantially the same asthe first displacement.
 14. The method of claim 2, wherein theprocessing the at least one second image is performed after displacingthe second sub-frame relative to the first sub-frame.
 15. A systemconfigured to for producing a motion video recording, the systemcomprising: an image processor configured to: receive a first imagecomprising at least a first sub-frame obtained from a first camera of afirst camera rig and a second sub-frame obtained from a second camera ofa first camera rig; receive a second image comprising at least a thirdsub-frame obtained from a first camera of a second camera rig and afourth sub-frame from a second camera of a second camera rig; calculateat least one first displacement between at least one first image pointin the first sub-frame and at least one corresponding second image pointin the second sub-frame; store the at least one first displacement in adisparity table; and process at least one of the third sub-frame and thefourth sub-frame in response to the disparity table.
 16. The system ofclaim 15, wherein the image processor is further configured to displacethe second sub-frame relative to the first sub-frame, and wherein atleast the first sub-frame and the fourth sub-frame form the secondimage.
 17. The system of claim 15, wherein the image processor isfurther configured to displace the fourth sub-frame relative to thethird sub-frame to create at least one second displacement.
 18. Thesystem of claim 15, wherein the image processor is further configured tochange the magnification of the second image in response to the at leastone first displacement until a second depth distance between a secondforeground point in the third sub-frame and a second background point inthe fourth sub-frame is substantially the same as a first depth distancebetween a first foreground point in the first sub-frame and a firstbackground point in the second sub-frame.
 19. The system of claim 15,wherein the image processor is further configured to blur all areas ofthe second image that lie outside a depth budget of the disparity table.